Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NVIDIA] Add Multi-graph Feature to NVIDIA Plugin #710

Merged
merged 27 commits into from
Sep 1, 2023
Merged

[NVIDIA] Add Multi-graph Feature to NVIDIA Plugin #710

merged 27 commits into from
Sep 1, 2023

Conversation

apavliuk55
Copy link
Contributor

@apavliuk55 apavliuk55 commented Aug 25, 2023 •
edited
Loading

Details:

  • Add splitting a model into multiple CUDA Graphs if not all of the operations support CUDA Graphs

Ticket:

  • 109090
  • 109091
  • 109092
  • 109093
  • 109095
  • 115700
  • 115701
  • 119112

@apavliuk55 apavliuk55 requested a review from a team as a code owner August 25, 2023 17:28
@github-actions github-actions bot added the category: NVIDIA plugin OpenVINO NVIDIA plugin label Aug 25, 2023
@apavliuk55 apavliuk55 force-pushed the feature/cuda-graphs-multi-graphs branch from 16cc846 to c8d6a4c Compare August 25, 2023 17:38
@apavliuk55 apavliuk55 changed the title [WIP] Add Multi-graph Feature to NVIDIA Plugin [WIP][NVIDIA] Add Multi-graph Feature to NVIDIA Plugin Aug 25, 2023
nkogteva
nkogteva reviewed Aug 28, 2023
View reviewed changes
modules/nvidia_plugin/src/cuda_graph_context.cpp Outdated Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_graph_context.cpp Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_graph_context.hpp Outdated Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_iexecution_delegator.hpp Outdated Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_infer_request.cpp Outdated Show resolved Hide resolved
modules/nvidia_plugin/src/ops/subgraph.hpp Outdated Show resolved Hide resolved
modules/nvidia_plugin/tests/unit/cuda_multi_graph_test.cpp Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_graph_topology_runner.cpp Outdated Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_graph_topology_runner.cpp Show resolved Hide resolved
modules/nvidia_plugin/src/cuda_graph_context.cpp Outdated Show resolved Hide resolved
@apavliuk55 apavliuk55 changed the title [WIP][NVIDIA] Add Multi-graph Feature to NVIDIA Plugin [NVIDIA] Add Multi-graph Feature to NVIDIA Plugin Aug 28, 2023
@apavliuk55 apavliuk55 force-pushed the feature/cuda-graphs-multi-graphs branch from c8d6a4c to 529b4a1 Compare August 29, 2023 15:31
@nkogteva
Copy link
Contributor

Please fix win build

10:16:40  [2023-08-30 08:16:33,530] [22664] build ie ERROR: C:\Jenkins\workspace\private-ci\ie\build-windows-vs2019\b\repos\openvino_contrib\modules\nvidia_plugin\src\cuda_simple_execution_delegator.hpp(72) : error C4716: 'ov::nvidia_gpu::SimpleExecutionDelegator::get_performance_counts': must return a value

@apavliuk55
Copy link
Contributor Author

Please fix win build

10:16:40  [2023-08-30 08:16:33,530] [22664] build ie ERROR: C:\Jenkins\workspace\private-ci\ie\build-windows-vs2019\b\repos\openvino_contrib\modules\nvidia_plugin\src\cuda_simple_execution_delegator.hpp(72) : error C4716: 'ov::nvidia_gpu::SimpleExecutionDelegator::get_performance_counts': must return a value

Done.
Also corrected the doxygen comments for SimpleExecutionDelegator.

nkogteva
nkogteva approved these changes Aug 30, 2023
View reviewed changes
Copy link
Contributor

@nkogteva nkogteva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, LGTM. There is only one open question about performance in case when number of cuda graphs too big.

modules/nvidia_plugin/src/cuda_graph_topology_runner.cpp Show resolved Hide resolved
Andrii Pavliuk added 20 commits August 30, 2023 17:48
[NVIDIA] Change CudaGraphTopologyRunner/SubGraph to aggregation from …
2f0468c 
…inheritance
[NVIDIA] Add new constructor to SubGraph
df7a759
[NVIDIA] Update CudaGraphContext to use vectors
fcc6096
[NVIDIA] Extract inputs/outputs to TensorMappingContext
590db80
[NVIDIA] Update SubGraph to use direct exec_sequence_ and std::shared…
05e13e7 
…_ptr<MemoryManager>
[NVIDIA] Add CudaGraphInfo and update CudaGraphContext to use them
9c0ed5d
[NVIDIA] Fix single-graph tests
2ff31ec
[NVIDIA] Add CudaMultiGraphTest test
b035b80
[NVIDIA] Add SubGraph::IsCudaGraphCompatible() cache
6af3486
[NVIDIA] Add execute_sequence/capture_sequence member functions to Pr…
cfba412 
…ofiler
[NVIDIA] Enable TensorIterator to use Profiler::execute_sequence()
e305986
[NVIDIA] Enable SubGraph to use Profiler::execute/capture_sequence()
26703db
[NVIDIA] Extract ITopologyRunner into the separate header
39baa30
[NVIDIA] Update tests to include cuda_eager_topology_runner.hpp
a493f44
[NVIDIA] Add IExecutionDelegator
8277039
[NVIDIA] Add cuda_perf_counts.hpp
3e4a839
[NVIDIA] Add SimpleExecutionDelegator class and use it when profiling…
3d820b1 
… is not needed
[NVIDIA] Update tests to use SimpleExecutionDelegator
d326be2
[NVIDIA] Add updateExecSequence() to TensorIteratorOp
5d27fc7
[NVIDIA] Update TensorIteratorOp::IsCudaGraphCompatible() to use SubG…
9395549 
…raph implementation
@apavliuk55 apavliuk55 force-pushed the feature/cuda-graphs-multi-graphs branch from fa3b2bb to 8f32a6b Compare August 30, 2023 14:48
@nkogteva nkogteva merged commit 394a8cf into openvinotoolkit:master Sep 1, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nkogteva nkogteva nkogteva approved these changes

Assignees
No one assigned
Labels
category: NVIDIA plugin OpenVINO NVIDIA plugin
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@apavliuk55 @nkogteva